Is Similarity Search Useful for High Dimensional Spaces?

Extended Abstract In recent years, multimedia content-based retrieval has become an important research problem. In order to provide effective and also efficient access to relevant data stored in large (often distributed) digital repositories, advanced software tools are necessary. Content-based retrieval works on the idea of abstracting the contents of an object, for example color or shape in the case of images, by so-called features – features are typically points in a high-dimensional vector space. Instead of determining the similarity of two objects based on their raw data, only the much smaller feature representations are used to estimate the objects' similarity. Given a reference (query) object represented by its features, similarity predicates are defined to retrieve a specific number of best cases or all objects satisfying a (distance) constraint. In this respect, we can distinguish between similarity range and nearest neighbor (NN) queries. Usefulness of NN-search. The dimensionality of a single feature type may range from small (4-9) to large (several hundreds). Frequently, it is not sufficient to query one feature in isolation. Rather, a query typically combines several feature types to better reflect the notion of similarity. Given this, the number of dimensions of combined features is well above 10. Beyer et al. [2] have recently questioned NN-search in high-dimensional vector spaces. The result of their theoretical analysis over a uniformly distributed data set is that NN-search as an implementation of similarity search is questionable if the number of dimensions is large, e.g. above 16. On the other hand, experiments with feature data of a large image database have shown that their finding must not always hold when applied to real data. It is still an open question how to find out whether the retrieved result of a query is of satisfactory quality for the user. Efficiency of NN-search. The feature approach to similarity search typically reduces the amount of searched data by orders of magnitude, e.g. from the tera-byte range to the giga-byte range. Since this reduction of data volume is not enough for large collections, researchers have proposed a number of methods which are mostly based on data-space partitioning. Index trees such as R-tree [5], X-tree [1], SR-tree [6], M-tree [3] or TV-tree [7] divide the data-space according to the distribution of data objects inserted or loaded into the tree. The main objective is to prune the search space in such a way that the NN can be …